Understanding Naturally Conveyed Explanations

One of the research goals of the field of human computer interaction is to make interacting with a computer as natural as interacting with a person. In pursuit of this goal, this thesis demonstrates a system that understands naturally conveyed explanations of mechanical devices in a way that traditional tools do not support. When using a traditional mechanical CAD program, for example, the designer formally specifies a spring connected to a block and must set parameters such as the length of the spring and its rest length. This is in contrast with the way a designer would describe the system to a colleague, by sketching the block and spring and saying "the spring pushes the block." Designers should not be forced to encode the behavior of the device in its parameters if the parameters are not the primary concern. This thesis presents a system, called ASSISTANCE, that understands naturally conveyed behavioral explanations. It demonstrates its understanding by generating a behavioral model for the device and answering questions posed by the designer about its model. ASSISTANCE allows a designer to sketch, gesture, and talk about a design in a familiar fashion, thus lowering the barriers to communication between the designer and the computer. Thesis Supervisor: Randall Davis Title: Professor of Computer Science and Engineering